feat(pipeline): add verify stage that runs the plan's proof and captures evidence - #18
Open
ci wants to merge 1 commit into
Open
feat(pipeline): add verify stage that runs the plan's proof and captures evidence#18ci wants to merge 1 commit into
ci wants to merge 1 commit into
Conversation
Owner
Author
|
This PR is part of a stack:
Created with jjpr |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The pipeline stopped at a clean review. Nothing ran the proof the plan promised (full gate, runtime behaviour, UI or simulator evidence), so the main agent had to do it by hand or trust the implementer's self-report.
Changes
verifystage afterreview: a fresh engine (--verify, defaultcodex:gpt-5.6-sol:high, envPIPELINE_VERIFY) receives the task, the plan (with its "Exact proof" section), and the implementation report, runs every proof command plus the repo gate, exercises runtime behaviour, and writes evidence files under<run>/evidence/.verify.json:status(pass|fail|blocked),proof(one entry per numbered proof item fromplan.json, validated for coverage, uniqueness, and range),checks(every command run; must be non-empty),evidence(files that must exist inside<run>/evidence/),notes, plus the tree id and proof snapshot it certified. Anything but a full pass halts with exit 3.reviewed_tree; verify refuses a different tree at start and halts if the committed tree changes during the stage (a verifier that "fixes" and commits cannot self-certify).runsends a run whose tree moved back to review, halts on a dirty workspace, and re-verifies when task/plan inputs changed since the verified run. Re-enteringrevieworverifyrewinds the completion pointer only after the workspace is known clean, so a halted rerun is retried byrun.plan.jsonproofis canonical; aplan.mdedit without a proof update halts verify until the list is changed orverify --accept-proofconfirms it. Baselines move to the last accepted or verified pair. A freshimplementafter review/verify resets its commit baseline.verifyrequires a completed review;runnow goes throughverifyby default (--untilaccepts it). Runs created before this change fall back to the default verify engine.summary.mdgains a Verification section listing checks and evidence paths; SKILL.md documents the stage, halts, permissions, and how to route a failed verification (fix and rerunverify, or amendtask.mdand rerunimplement+review).Tests
uvx pytest -q, three CLI behaviours), three evidence files written, statusverified, summary section rendered.fail, onblocked, onpasswith a failed check; completion on a clean pass; ordering guard when review is not complete;STAGESorder.verified.--accept-proof; baseline refresh;runrewind on tree change, halt on dirt, re-verify on input change; review re-entry rewind; summary staleness for tree, proof, and input changes.autoreview --reviewers codex:gpt-5.6-sol:xhigh,grok:grok-4.6:xhigh: five rounds (the cap). Rounds 1-4 findings all fixed. Round 5 raised three P2/P1 items (dirty workspace after verify treated as done byrun; summary not stale on task/plan prose edits; implement baseline retained across a post-verify rerun); all three fixed after the cap without a further review round. Grok reported clean in rounds 2-5.